Note to reviewer: Let's assume we can save client secrets as server options for now. The credentials are part of USER MAPPING in this PR #255

▶️ How users can create REST catalogs

CREATE SERVER my_polaris TYPE 'rest'
  FOREIGN DATA WRAPPER iceberg_catalog
  OPTIONS (rest_endpoint 'http://polaris:8181',
           client_id '...', client_secret '...',
           location_prefix 's3://my-bucket/warehouse');

CREATE TABLE t (a int) USING iceberg WITH (catalog = 'my_polaris');

A server with no options is also valid — all settings fall back to GUCs. This allows creating a named handle for organizational purposes while relying entirely on the global GUC configuration.

▶️ Accepted Server Options

The iceberg_catalog_validator accepts the following server-level options (all defined in iceberg_catalog_option_descs[]):

Credentials (client_id, client_secret) are currently stored as server options. USER MAPPING support is in a follow-up PR.

▶️ catalog_name Server Option

The catalog_name option on an iceberg_catalog server specifies the catalog prefix used in REST API URLs (opts->catalogName).

At CREATE TABLE time (read-only tables): if the table does not specify catalog_name, it inherits the server's catalog_name option (via ResolveRestCatalogOptions) or defaults to get_database_name(MyDatabaseId). The resolved value is baked into the table's ftoptions.

At runtime (GetRestCatalogName):

• Writable tables always return get_database_name(MyDatabaseId). Server and table catalog_name options are ignored. This prevents a subsequent ALTER SERVER ... ADD/SET catalog_name from silently re-routing an existing writable table.
• Read-only tables return the table-level catalog_name from ftoptions. If missing, error (should never happen after CREATE TABLE inheritance).

Writable tables cannot be created on a server with catalog_name set — enforced at CREATE TABLE time.

▶️ Extension Upgrade Script

The upgrade script creates:

1. iceberg_catalog FDW — a handler-less FDW with a validator function (iceberg_catalog_validator) that accepts server-level options.
2. GRANT USAGE ON FOREIGN DATA WRAPPER iceberg_catalog TO lake_write — so non-superusers with lake_write can create catalog servers.
3. Three built-in catalog servers — pg_lake_postgres_catalog (TYPE 'postgres'), pg_lake_object_store_catalog (TYPE 'object_store'), and pg_lake_rest_catalog (TYPE 'rest'). These are extension-owned structural anchors backed by actual pg_foreign_server rows, but carry no options — all configuration comes from GUCs.
4. GRANT USAGE ON FOREIGN SERVER ... TO lake_write — for each built-in server, so non-superusers with lake_write can create tables against the built-in catalogs.

The reserved user-facing catalog names postgres, object_store, and rest are mapped internally to these built-in server names via ResolveCatalogServerName. A pre-flight check in the upgrade script prevents conflicts if a user already has a server with one of the built-in long names.

▶️ DDL Protection (ValidateIcebergCatalogServerDDL)

A single ProcessUtility hook validates all DDL on iceberg_catalog servers, the iceberg_catalog FDW itself, and extension-membership changes that affect any of these objects. The handler is registered in pg_lake_iceberg/src/init.c.

Two layers of protection for server DDL:

1. Short reserved names ('postgres', 'object_store', 'rest') — the user-facing catalog= values. CREATE SERVER and RENAME TO these names are blocked so users can't shadow the built-in catalogs.
2. Built-in long server names (pg_lake_postgres_catalog, etc.) — the pre-created anchors. Outside of CREATE/ALTER EXTENSION, CREATE/ALTER/RENAME/OWNER on them is blocked entirely so they remain pure structural anchors with all configuration in GUCs. DROP is blocked by PostgreSQL core via extension membership (pg_depend deptype 'e').

All protection checks are skipped during CREATE EXTENSION / ALTER EXTENSION UPDATE. The ALTER EXTENSION DROP and ALTER FOREIGN DATA WRAPPER guards are defense-in-depth against superuser misuse: both require superuser privilege, but detaching the extension edge or swapping the validator would silently weaken the protection model.

▶️ Single-Catalog Transaction Constraint

Modifying tables from different REST catalogs in the same transaction is rejected at statement time — before any Parquet data is written to S3. The first mutation binds the transaction to its REST catalog; any subsequent mutation targeting a different catalog raises ERRCODE_FEATURE_NOT_SUPPORTED immediately.

This applies to all mutation entry points: INSERT, UPDATE, DELETE, and TRUNCATE, as well as DDL paths (CREATE TABLE / DROP TABLE) which reach the same protection indirectly.

▶️ Backward Compatibility

• CREATE TABLE ... WITH (catalog = 'rest') continues to work. The catalog option value 'rest' is mapped internally to the built-in pg_lake_rest_catalog server, whose configuration comes entirely from GUCs (case-insensitive: 'REST', 'rEst', etc. all resolve to the same built-in server).
• All existing GUCs remain functional and serve as fallback defaults for both built-in and user-created catalogs.
• No changes to postgres or object_store catalog behavior.
• A pre-existing CREATE SERVER postgres FOREIGN DATA WRAPPER postgres_fdw does not conflict with pg_lake's built-in postgres catalog, since the user-facing short name 'postgres' maps internally to pg_lake_postgres_catalog.

Implementation Internals

▶️ Connection Resolution

ResolveRestCatalogOptions(catalog) is the single entry point for building a RestCatalogOptions. GetRestCatalogOptionsForRelation(relationId) reads the table's catalog option and delegates to it.

All catalogs — built-in and user-created — go through the same resolution path:

1. ResolveRestCatalogOptions(catalog) — calls ResolveCatalogServerName(catalog) to map short reserved names ('rest' → pg_lake_rest_catalog, 'postgres' → pg_lake_postgres_catalog, 'object_store' → pg_lake_object_store_catalog); user-created server names pass through unchanged. Then delegates to BuildRestCatalogOptionsFromServer(serverName, catalog).
2. BuildRestCatalogOptionsFromServer(serverName, userVisibleCatalog) — looks up the server in pg_foreign_server, initializes all fields from GUC defaults via ApplyGUCDefaults, then overrides with any options explicitly set on the server via ApplyServerOptionOverrides (driven by iceberg_catalog_option_descs[]). The userVisibleCatalog (the short name the user typed) is stored in opts->catalog so error messages, the token cache key, and the cross-catalog DML check stay in user-facing terms.
3. ValidateRestCatalogOptions — errors if rest_endpoint is still unset after both sources.

Since ALTER SERVER OPTIONS is blocked on the built-in servers, their option set is always empty and the GUC defaults survive untouched — achieving the "GUCs-only built-in REST" behavior through a single code path.

catalog_name on the server is stored in opts->catalogName when present, but it is not defaulted to get_database_name() inside resolution. For read-only tables, the database-name default is applied at CREATE TABLE time: if the table does not specify catalog_name, create_table.c inherits the server's catalog_name option (if set) or defaults to get_database_name(MyDatabaseId), then bakes the result into the table's ftoptions. Writable tables ignore server catalog_name at runtime (see above).

▶️ Option Validation

Server options are defined once in iceberg_catalog_option_descs[] — the single source of truth for the whitelist (FindCatalogOptionDesc), the user-facing hint (GetValidCatalogOptionsHint), and the option-to-struct applier (ApplyCatalogOptionValue). Adding or removing an option requires changing only this table.

FDW option names and auth type values are compared case-insensitively (pg_strcasecmp). Server names remain case-sensitive (PostgreSQL's GetForeignServerByName is case-sensitive), but the reserved built-in names are checked case-insensitively.

At CREATE/ALTER SERVER time, ValidateCatalogOptionValue enforces:

• rest_endpoint, oauth_endpoint, location_prefix: non-empty and must contain :// (URI scheme required)
• catalog_name, client_id, client_secret, scope: non-empty when present
• rest_auth_type: must be 'oauth2', 'default' (alias for oauth2), or 'horizon'
• enable_vended_credentials: valid boolean

The validator hint string is built lazily from iceberg_catalog_option_descs[] and cached in TopMemoryContext so it survives transaction aborts on the error path (a per-statement context would leave a dangling pointer on the second invalid-option failure in the same backend).

▶️ Per-Catalog Token Cache

The old code used a single global token (RestCatalogAccessToken). With multiple catalogs potentially using different credentials, the token cache is now a hash table (RestCatalogTokenCache) keyed by the iceberg_catalog server OID (opts->serverOid). Each catalog gets its own cached OAuth token with independent expiry tracking.

The hash table and token strings are allocated in a dedicated RestTokenCacheCtx memory context (under CacheMemoryContext), keeping the cache memory isolated.

Invalidation on server DDL: a syscache invalidation callback (InvalidateRestTokenCache) is registered once per backend via CacheRegisterSyscacheCallback(FOREIGNSERVEROID, ...). Any ALTER SERVER or DROP SERVER resets the entire token cache (MemoryContextReset(RestTokenCacheCtx)) so stale credentials are never reused in open sessions. The cache is rebuilt lazily on the next token lookup. TokenCacheCallbackRegistered ensures the callback is registered exactly once (the syscache callback array is fixed-size).

SendRequestToRestCatalog implements its own retry loop (up to 3 retries) with a ClassifyRestCatalogRequestRetry classifier that determines whether to back off, refresh auth, or stop. When a 419 (token expired) response is received, the retry logic force-refreshes the token via GetRestCatalogAccessToken(opts, true) and patches the Authorization header in-place via UpdateAuthorizationHeader. The opts parameter is passed directly to SendRequestToRestCatalog, avoiding any global state. Pass opts = NULL for the token-fetch request itself to avoid recursion.

▶️ Single-Catalog Transaction Constraint (Internals)

BindRelationToXactRestCatalog(relationId) is called from every entry point that can mutate REST-backed iceberg tables:

• postgresBeginForeignModify() — row-by-row DML
• AddQueryResultToTable() — INSERT...SELECT and COPY...FROM pushdown
• postgresExecForeignTruncate() — TRUNCATE

All per-transaction REST catalog state is grouped in a single PgLakeXactRestCatalogContext struct (static PgLakeXactRestCatalog), which contains:

• requestsHash — hash table of per-table REST catalog requests
• commitContext — pre-allocated 1MB memory context for XACT_COMMIT
• catalogOpts — deep-copied RestCatalogOptions for the single catalog

On the first REST-backed write of the transaction, BindRelationToXactRestCatalog pre-resolves the relation's full RestCatalogOptions and deep-copies them into TopTransactionContext (PgLakeXactRestCatalog->catalogOpts). This happens before any Parquet data is written to S3. Subsequent writes in the same transaction must target the same catalog (compared by server OID — opts->serverOid — so case variations like 'rest' and 'REST' correctly collapse to the same built-in server); otherwise ERRCODE_FEATURE_NOT_SUPPORTED is raised immediately.

DDL paths (CREATE TABLE / DROP TABLE) reach the same protection indirectly via RecordRestCatalogRequestInTx(), which is invoked synchronously at statement time from the utility hook. That function retains a belt-and-suspenders cross-catalog check for any code path that forgets to bind at statement time.

At XACT_EVENT_COMMIT time, PostAllRestCatalogRequests uses the pre-resolved PgLakeXactRestCatalog->catalogOpts to build URLs and authentication headers. This avoids syscache lookups during commit, which are forbidden (AssertCouldGetRelation fires during TRANS_COMMIT state).

When an iceberg table is created on any catalog server — user-created or built-in — RecordIcebergCatalogServerDependency records a DEPENDENCY_NORMAL edge from the table to the server in pg_depend. For built-in catalogs, the short reserved name (e.g. 'rest') is mapped to the built-in server name (pg_lake_rest_catalog) via ResolveCatalogServerName. This ensures DROP SERVER is blocked while dependent tables exist, and DROP EXTENSION CASCADE cleans up tables that depend on the built-in servers.

▶️ Catalog Type Helpers (pg_lake_engine/src/utils/catalog_type.c)

• IsRestCatalog(catalog) — unified check that returns true for the literal 'rest' (case-insensitive) or any iceberg_catalog server whose TYPE is 'rest'. Internal built-in server names are explicitly rejected (they are implementation details, not user-facing catalog values). Used by the GUC check hook for default_catalog to accept user-created server names.

• IsCatalogOwnedByExtension(catalog) — returns true for the short reserved names 'rest', 'object_store', or 'postgres' (all case-insensitive via pg_strcasecmp). Used by the DDL protection handler, dependency recording, and ACL checks.

• IsBuiltinCatalogServerName(serverName) — returns true for the long built-in server names (pg_lake_rest_catalog, pg_lake_postgres_catalog, pg_lake_object_store_catalog), case-insensitive. The long-name counterpart to IsCatalogOwnedByExtension. Used by the DDL protection hook to lock down the extension's structural anchors.

• ResolveCatalogServerName(catalog) — maps user-facing short names to the corresponding built-in server name; passes user-created server names through unchanged.

Fixes part of #230

Checklist

• Probably need to intercept create server commands that use variations of case-insensitive "postgres", "rest" and "object_store" reserved literals.
• I have tested my changes and added tests if necessary
• I updated documentation if needed
• I confirm that all my commits are signed off (DCO)